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AN  EVALUATION  STUDY  OF  OFF-THE-SHELF 
PATIENT  CLASSIFICATION  SYSTEMS 


INTRODUCTION 

Purposa 

This  study  was  conducted  to  determine  if  any  commercially 
available,  off-the-shelf  patient  classification  systems  (PCS) 
would  meet  the  functional  requirements  for  the  Department  of 
Defense's  (DOD)  nursing  management  information  needs. 
Additionally,  information  about  the  commercial  system 
methodologies  for  patient  classification  and  staffing  were 
identified  and  considered  in  the  development  of  other  options  for 
the  revision  of  the  Workload  Management  System  for  Nursing 
(WMSN)  . 


Background 

The  Army,  Navy,  and  Air  Force  Nurse  Corps  currently  utilize 
WMSN  for  nursing  resource  management.  The  basic  research  for 
WMSN  was  conducted  from  1977  to  1985  (Sherrod,  Rauch,  &  Twist, 
1981;  Misener,  Frelin,  &  Twist,  1983;  Vail,  1986).  In  August 
1991,  in  the  interest  of  maintaining  the  clinical  and 
methodological  currency  of  WMSN,  Nursing  Studies  Branch,  the  U.S. 
Army  Health  Care  Studies  and  Clinical  Investigation  Activity  was 
directed  by  the  Office  of  the  Assistant  Secretary  of  Defense, 
Health  Affairs  (OASD[HA])  to  consider  all  reasonable  alternatives 
in  revising  WMSN. 

The  process  of  developing  research  study  concepts  and 
evaluating  the  functionality  of  commercial  PCS  systems  had  to  be 
accomplished  in  a  relatively  short  time.  Members  of  the  Nursing 
Studies  Branch  (NSB)  identified  four  options:  two  approaches 
using  the  current  system's  basis  of  acuity  and  previous  research, 
a  multidimensional  approach  to  acuity  more  consistent  with 
professional  practice,  and  the  purchase  of  an  off-the-shelf 
system. 

Although  an  evaluation  of  commercial  systems  can  examine 
many  characteristics,  functionality  is  usually  given  the  highest 
priority.  Functionality  encompasses  not  only  the  range  of 
functions  performed,  but  the  specific  way  the  system  accomplishes 
those  functions  (Pivnicny  &  Carmody,  1989) .  Before  committing  to 
expensive,  time-consuming  testing,  it  is  essential  to  identify 
the  systems  that  will  produce  the  desired  outcomes  in  a 
satisfactory  manner.  The  evaluation  model  used  in  this  study 
provided  a  framework  for  functional  information  that  had 
relevance  to  DOD  medical  treatment  facilities. 

Very  often,  organizations  begin  testing  systems  without  a 
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clear  understanding  of  their  requirements.  In  addition, 
marketing  information  often  overstates  a  system's  capabilities 
and  understates  its  limitations.  Conflicting  definitions  of  what 
should  be  common  terms  further  complicates  making  comparisons. 

All  these  factors  can  significantly  delay  putting  a  new  system 
into  the  hands  of  the  nursing  staff.  However,  by  clearly 
establishing  the  primary  purpose  of  classification,  selection 
criteria  can  be  used  to  identify  suitable  tools  for  further 
cons idera t ion . 


OBJECTIVES 

A  3-phase  plan  was  developed  to  expedite  identification  of 
suitable  systems  using  readily  available  information. 
Respectively,  the  objectives  were  to:  (a)  identify  and  collect 
information  on  commercially  available,  off-the-shelf  PCS's,  (b) 
organize  pertinent  information  using  an  adapted  evaluation  model, 
and  (c)  identify  the  PCS's  meeting  predetermined  selection 
criteria. 

METHODOLOGY 

Overview 

Eighty-nine  potential  PCS  sources  were  identified  from  lists 
in  nursing  administration  and  hospital  automation  references 
("Directory  of  Consultants,"  1990;  Roland  &  Roland,  1989; 
"Software  Guide,"  1991).  A  data  collection  form  was  developed  to 
guide  the  telephone  interviews.  Written  materials  on  PCS 
methodology  were  also  requested.  Information  was  sorted  using  an 
adapted  model  which  organized  data  according  to  system  content, 
input,  process  and  product.  A  matrix  of  the  selection  criteria 
identified  suitable  PCS's. 


Procedures 
Phase  I 

PCS  sources  from  three  lists  were  compiled  from  the 
literature.  The  majority  of  the  PCS  sources  were  consultant  or 
information  systems  companies.  All  were  contacted  using  the 
telephone  numbers  provided  in  the  listings.  A  data  collection 
form  guided  the  telephone  interviews  conducted  by  three  NSB  staff 
and  assured  consistency  in  the  interview  process  (see  Appendix 
A) .  Written  information  on  the  PCS  methodology  was  requested  and 
follow-up  calls  were  made  as  needed. 

The  main  objective  of  the  initial  telephone  call  was  to 
identify  a  point  of  contact  (POC)  with  knowledge  of  the  PCS 
methodology.  Most  firms  referred  the  inquiries  to  registered 
nurses  on  their  staff  who  functioned  as  members  of  the  design, 
research  and/or  implementation  teams.  When  this  was  not 
possible,  questions  were  answered  by  others,  usually  marketing 
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personnel.  Although  the  information  received  from  marketing 
personnel  may  not  be  totally  accurate  or  complete,  it  was  not 
unreasonable  to  expect  a  company's  representative  to  provide 
accurate  product  information  to  potential  clients.  This  was 
considered  an  acceptable  limitation,  as  the  purpose  of  the  study 
was  to  identify  functionally  suitable  systems  for  further  review, 
not  to  make  a  final  selection.  In  some  instances,  requests  for 
return  telephone  calls  and/or  written  materials  were  ignored. 
After  a  reasonable  number  of  attempts  had  been  made,  fourteen 
sources  were  dropped  from  the  study  due  to  lack  of  response  from 
the  companies. 

Fifty-seven  sources  were  eliminated  during  the  course  of  the 
study  using  other  rejection  criteria.  Some  sources  did  not  have 
a  PCS  methodology.  These  included  nursing  information  systems 
without  PCS  modules,  software  programs  that  automate  a  hospital's 
existing  PCS,  management  applications  of  acuity  data,  and  firms 
that  only  provided  consultative  services.  Eight  of  the 
eliminated  sources  had  PCS's  used  exclusively  in  limited 
specialty  areas,  such  as  Emergency  Departments  or  long-term  care 
facilities.  Other  sources  had  PCS's  that  were  one  module  of  an 
extensive  automated  system  that  could  not  function  as  a  stand¬ 
alone  system. 

Product  information  was  provided  by  fifty-nine  of  the 
sources.  Packets  were  made  combining  the  completed  telephone 
data  collection  forms  and  the  mailed  infomnation.  Code  numbers 
were  assigned  to  each  packet  and  used  for  the  remainder  of  the 
study.  This  was  done  to  reduce  selection  bias  by  the  NSB 
researchers  in  the  event  several  potentially  suitable  systems 
were  identified. 

There  was  an  additional  benefit  of  the  study.  Methodologies 
and  presentation  formats  of  interest  to  the  other  WMSN  revision 
options  were  also  identified  and  shared  with  the  appropriate 
nurse  researchers.  These  included  non-time  based  acuity 
methodologies,  customized  PCS  and  staffing  systems,  specialty 
nursing  task  lists,  report  formats,  and  marketing  layouts. 

Phase  II 

The  context- input-process-product  (CIPP)  model  is  an 
evaluation  tool  proposed  by  Stuff lebeam  (1987)  for  use  with 
educational  programs.  Context,  input,  process  and  product  are 
the  four  parameters  evaluated  in  the  model  (Stuff lebeam  et  al, 
1971) .  Information  about  objectives,  needs  and  expectations  is 
obtained  through  context  evaluation.  Input  evaluation  yields 
information  regarding  procedural  designs  as  well  as  strengths  and 
limitations.  Process  evaluation  provides  information  regarding 
implementation  and  monitoring.  Product  evaluation  allows 
decision  makers  to  assess  information  about  results,  reactions, 
and  deficiencies. 
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After  synthesizing  information  from  the  literature  regarding 
patient  classification  systems,  the  investigators  constructed  a 
framework  comprised  of  four  parameters:  operating  context, 
system  capabilities,  implementation  design  and  outcomes. 

Specific  data  requirements  within  each  parameter  are  presented  in 
Appendix  B. 

The  final  draft  of  the  CIPP  evaluation  matrix  was  reviewed 
and  approved  by  NSB.  The  matrix  was  entered  into  the  word 
processor  to  speed  up  the  evaluation  process  and  improve 
legibility.  Although  the  matrix  proved  somewhat  awkward  and 
redundant,  redesign  would  have  delayed  rather  than  improved  data 
entry . 

In  Phase  II,  one  nurse  researcher  started  a  CIPP  evaluation 
matrix  for  each  PCS  source  using  the  information  from  the 
telephone  data  collection  form  and  the  mailed  information. 

Sources  were  eliminated  from  the  study  as  soon  as  they  were 
identified  as  meeting  one  of  the  rejection  criteria  discussed 
previously.  This  occurred  at  various  stages  of  the  study.  Some 
sources  remained  in  the  study  through  Phase  III,  while  others 
were  eliminated  after  the  first  telephone  call. 

Phas*  III 

Selection  criteria  for  a  major  system  requires  clarity  about 
the  strategic  goals  of  the  organization  as  well  as  the  needs  of 
customers  at  all  levels.  Therefore,  once  information  about  the 
systems  had  been  sorted,  criteria  were  developed  to  identify 
systems  meeting  functional  requirements.  In  November  and 
December,  the  selection  criteria  were  reviewed  by  the  Nursing 
Consultant,  Office  of  the  Surgeon  General  and  the  Assistant 
Chief,  Army  Nurse  Corps.  The  criteria  were  approved  in  mid- 
December  and  required:  (a)  reliable  &  valid  tool(s)  for  the 
seven  clinical  areas  currently  using  WMSN,  (b)  completed  tool(s}, 
or  minimal  adaptation  required,  (c)  standardized  categories  used 
across  clinical  specialties,  (d)  information  for  rating  patient 
accessible  in  patient  record,  (e)  in  use  in  multiple  sizes  and 
types  of  hospitals,  (f)  an  audit,  or  interrater  reliability 
system,  and  (g)  a  predictive  daily  staffing  system. 

FINDINGS 

Eighteen  sources  were  identified  as  having  commercially 
available  PCS's  and  were  evaluated  using  the  CIPP  matrix.  (See 
Appendix  C  for  an  alphabetical  listing.)  Four  sources  were 
subsequently  deleted  when  they  were  found  to  provide  duplicate 
tools.  Table  1  displays  the  application  of  the  selection 
criteria  to  the  14  final  PCS  tools. 
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Table  1 


Patient  Claaflif ication  Tools  Sorted  Aaeinet  Selection  Criteria 


PCS 

Code 

No. 

Used  in 
7 

clini¬ 

cal 

areas 

Same 

cate¬ 

gories 

all 

areas 

Needs 

minimalor 

no 

modifi¬ 

cation 

All 

info 

in 

chart 

for 

audit 

Multi¬ 
ple  MTP 
sites 
and 
types 

Audit 

System 

staff 

System 

01 

X 

X 

X 

X 

X 

X 

X 

04 

X 

M 

N 

X 

X 

X 

X 

09 

X 

N 

X 

N 

X 

X 

X 

15 

N 

M 

X 

X 

X 

X 

X 

18 

X 

N 

N 

X 

X 

X 

X 

20 

X 

X 

X 

N 

N 

X 

X 

21 

X 

X 

M 

X 

X 

X 

X 

23 

X 

X 

X 

X 

X 

X 

X 

24 

M 

X 

X 

X 

X 

X 

X 

34 

X 

N 

M 

X 

X 

X 

X 

35 

X 

N 

N 

N 

X 

X 

X 

40 

N 

X 

N 

X 

X 

X 

X 

43 

X 

N 

X 

X 

N 

X 

X 

53 

X 

X 

N 

X 

X 

X 

X 

Note.  X 

*  meets 

criteria; 

N  «  does  not 

meet  criteria. 

Two  of  the  14  systems  met  all  selection  criteria:  Van  Slyck 
(01)  and  Medicus  (23).  The  two  systems  have  many  common 
characteristics.  Both  are  reported  by  their  companies  as  well- 
researched,  valid  and  reliable  PCS  tools  reflecting  current 
nursing  practice.  Developed  in  the  1970' s,  both  tools  are  used 
nationwide  in  many  teaching  and  nonteaching  acute  care  facilities 
of  various  sizes.  Factor  evaluative,  or  checklist-type  tools, 
they  use  standard  categories  across  all  units.  The  acuity  values 
are  weights,  not  time.  Interrater  reliability  is  determined  from 
information  in  the  medical  record. 

The  number  of  patients  per  acuity  category  is  reported  and 
therefore  available  for  development  of  acuity-based  manpower 
standards.  Translating  acuity  weights  to  full-time  equivalents 
by  the  staffing  systems  requires  some  customization  by  the 
companies  to  identify  the  needs  and  philosophy  of  the  facility  or 
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hospital  system.  The  staffing  systems  project  staffing 
requirements  as  nvunbers  of  each  skill  mix  per  shift. 


Additionally,  both  companies  have  other  management 
applications  that  use  the  acuity  and  staffing  data.  The  cost  of 
the  two  systems  Is  flexible  and  dependent  on  the  amount  and  type 
of  support  requested  from  the  companies.  Implementation  time 
appears  about  the  same  for  both  systems. 

Despite  the  slmlllarltles  between  the  two  systems  and  the 
fact  that  they  both  meet  the  preestabllshed  selection  criteria, 
significant  differences  exist  between  the  Medlcus  and  Van  Slyck 
systems.  Table  2  details  the  differences. 

Table  2 

Differences  Between  the  Medicua  PCS  and  the  Van  Slvck  PCS 


Criteria 
Basis  of  acuity 

Nunnber  of  indicators 

Time/ frequency  of  acuity 
rating 

Tool  modification 

Face  validity  for 
nursing  staff 

Number  of  categories 

Staffing  system 

Input  used  to  determine 
staffing 

Number  of  hospitals 
using  the  tool 

Automation 

Access  to  civilian  data 
base 

Maintenance  of  system 


Medicus 

Amount  &  complexity  of 
care 

36 

Daily 

None  required 

Total  nursing  domain  not 
shown 

6 

Projects  staffing  3 
shifts  based  on  planned 
care 

Lirect  and  indirect 
time,  operating 
constraints 


300 


Multiple  options 
available 

Yes,  annual  reports 

Comprehensive  support  & 
maintenance 


Van  Slvck 

Risk,  complexity,  skill 
level  &  time 

70 

Every  shift 
Minimal 

Total  nursing  domain  is 
shown 
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Projects  staffing  1  to  3 
shifts  based  on  actual 
care 

Direct  and  indirect 

time,  short  length  of 

stay  patients, 

out pat ient 8 ,  oper at ing 

constraints 

70 

Manual  only 
No 

Annual  updates 


The  major  underlying  difference  between  the  two  systems,  the 
use  of  actual  versus  prospective  workload,  is  the  most  critical 
one.  This  and  the  other  differences  between  the  two  PCS's 
provide  a  mechanism  for  selecting  a  system  to  replace  the  current 
WMSN,  if  an  off-the-shelf  option  is  deemed  acceptable  to  DOD. 


DISCUSSION 

The  sense  of  urgency  felt  by  users  and  top  management  to 
replace  an  obsolete  system  is  understandable.  "If  problems  with 
an  existing  system  are  serious  enough  to  justify  spending  tens  of 
thousands  of  dollars  to  find  a  new  system  .  .  .  how  can  a 
hospital  spend  years  waiting  for  a  new  system  to  be  in  place?" 
(Doyle,  1990) .  The  process  of  identifying  a  system  that  meets 
organizational  needs  is  necessarily  time  consuming. 

However,  there  are  strategies  for  meeting  the  goals  of  the 
search  for  a  system  and  reducing  time  delays.  First,  senior 
management  must  define  their  expectations.  The  requirements  of 
the  system  should  be  derived  from  the  organization's  strategic 
plans  (Doyle,  1990) .  Selection  criteria  can  then  be  developed 
that  identify  the  purpose  of  the  system  and  set  priorities  on 
customers'  needs  in  accomplishing  that  purpose.  The  next  major 
step  is  to  identify  systems  for  thorough  review  and  pilot 
testing.  Missteps  can  result  in  expensive  delays  by  implementing 
and  testing  systems  that  do  not  meet  functional  requirements. 

CONCLUSIONS 

Adapting  the  Stuff lebeam  CIPP  model  resulted  in 
identification  of  relevant  data  in  a  systematic  manner, 
facilitating  the  judgement  process.  Clearly  identifying  systems 
that  meet  functionality  requirements  from  readily  available 
information  reduced  both  the  time  and  cost  of  the  selection 
process . 


RECOMMENDATIONS 

Although  the  model  clarified  the  selection  process,  more 
information  was  obtained  and  organized  initially  than  was 
required.  As  the  philosophy  and  needs  relative  to  patient 
classification  were  clarified,  some  information  was  no  longer 
required  to  evaluate  the  systems. 

Examining  system  functionality  does  not  eliminate  the  need 
for  further  evaluation.  Other  aspects  of  purchasing  a  system 
must  be  explored  to  insure  valid  performance  in  military  practice 
settings.  The  findings  of  this  study  were  briefed  to  the  Chief, 
Army  Nurse  Corps,  OASD(HA) ,  and  the  Tri-service  WMSN  Working 
Group  for  further  consideration  at  the  corporate  level. 
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If  purchasing  a  system  Is  deemed  by  nursing  leadership  as 
the  most  cost  effective  solution  to  revising  WMSN,  the 
differences  between  the  two  systems  provide  a  mechanism  for  final 
selection.  If  provided  strategic  planning  information,  a 
multidisciplinary  task  force  could  weight  the  desirable 
characteristics  to  identify  a  system  for  pilot  testing. 
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Appendix  A 


Patient  Classification  fivsten  Survey  Worksheet 

Date/ Initials _ 

PCS  Name _ 

Organization _ 

Address  _  _ 


POC  &  Phone  No. 


General  Description  of  PCS; 

1.  Type  of  PCS?  _ factor  evaluative  _ prototype 


2.  Patient  acuity  categories?  _ ^yes  _ no 

_ no .  of  categories 


3.  Same  categories  used  in  all  clinical  areas?  _ yes  _ no 


4.  Clinical  areas?  (Mark  which  have  separate  tool) 


crit  care 
medsurg 
outpt 
OB/GYN 


ED 

NBN 

■PACU 

psych 


L&D 

NICU 

Peds 

sameday  surg 


5.  Is  the  tool  used  to  collect  data 


6.  How  often  is  the  tool  completed? 


retrospect i ve ly 

concurrently 

prospectively 

once/ shift  _ once/day 
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7.  How  is  IRR  done? 


8.  All  IRR-required  info  in  patient  record? 

9.  Tool's  uses?  _ pt  acuity 

_ daily  staffing  _ staffing  standard 

_ ^productivity  monitoring  _ QA 

_ costing  nursing  services  _ billing 

10.  Easy  for  staff  to  use? 

How  long  does  it  take  to  fill  out?  _ 

Manual _  Automated _ 

Acceptance  by  staff?  (explain) 

11.  Strengths  of  the  PCS? 

12.  Weaknesses  of  the  PCS? 

B.  Methodolocfv 

1.  Who  developed  the  tool?  _ corporate  _ homegrown 

2.  When  was  it  developed/ implemented/updated? 

3 .  Sample  size? 

4.  How  were  tasks  or  categories  defined? 

5.  How  were  times  determined?  (task  and/or  category) 

_ ^time  studies  _ experts  _ self-report  _ industry 

standards 
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6.  Is  complexity  of  tasks  addressed?  (must  be  done  by  RN, 
initiated  by  RN  rather  then  Dr's  order,  etc) 


7.  Does  the  tool  give  credit  for  staff  time  spent  assessing, 
evaluating,  and  integrating  patient  care?  (cognitive  component) 


8.  Are  staffing  requirements  based  on 

_ direct  hours  _ indirect  hours  _ unit  mgmt  hours? 


9.  Reliability  and  validity  studies? 


C.  Information  available?  fand  costs) 

_ copy  tool  _ user  instructions 

_ task  list  _ ^definitions 

_ category  descriptions 

_ ^manuals  _ ^reports 

_ articles,  publications 


study  data 
times 
times 
software 


Cj _ Show  and  Tell 

1.  What  type  of  hospitals  use  the  tool?  _ no. beds 

_ public  _ ^private 

2.  Hospitals  in  this  area  using  the  tool?  (and  POC/phone  no.) 


3.  Do  you  perform  demos? 
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Appendix  B 

CIPP  Bvaluatien  Matrix 


I.  PCS  Tool  Code  Humber: 

II.  Operating  Context:  Objectives  associated  with  system, 
intended  uses  (needs) ,  opportunities 

a.  Purpose 

b.  Setting 

1.  acute  care  hospitals 

2.  small,  medium  and  large 

3.  type  of  clinical  units 

a)  current  7  areas:  med>surg,  OBGYN,  peds,  NICU,  crit  care, 
psych,  NBN 

b)  others  (L&D,  PACU,  OR,  SDS,  ED,  AMB,  etc) 

c)  combined  units,  inpatient  units  seeing  outpatients,  etc 

e.  Historical  information 

1 .  year  developed 

2.  length  of  time  in  use 

3.  scope  (number  and  type  of  MTFs) 

d.  Intended  and  realised  infonaation  needs  generated 

1 .  patient  acuity 

2.  scheduling  by  day,  shift 

3 .  productivity  monitoring 

4 .  audit  procedure 

5.  quality  assurance 

6 .  costing 

7.  facility  staffing  requirements 

8.  national  data  base 

9 .  other 
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III.  Input  or  SystoB  Capabilitios:  strategies  and  actions  for 
accomplishing  objectives 

a.  Design  of  instrument 

1.  theoretical  framework  or  rationale 

a)  factor  evaluative  or  prototype 

b)  basis  for  acuity  (pt  need,  nsg  task,  time,  etc) 

2.  content  (behavioral  tasks;  cognitive  tasks) 

3 .  format 

b.  Research 

1 .  methodology 

a)  actual  or  estimated  times 

2 .  sample 

3 .  data  analysis 

4.  reliability  and  validity 

c.  Prooedures  for  use 

1.  frequency  of  data  collection,  classifications 

2.  workload  measured:  actual  or  projected 

3.  additional  forms,  documentation  required 

4.  unit  personnel  involvement 

5.  time  to  complete  instrument 

6.  source  of  patient  information 

d .  Resource  requirements 

1.  personnel 

2 .  automation  or  other  equipment 

3 .  training 

4.  finances 

5 .  other 
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IV.  Process  or  Inplofflentation  Design:  procedures  or  strategies 
for  achievement  of  objectives,  as  well  as  design  defects 

a.  Implementation  method 

1.  Customized  items 

a)  indicators 

b)  times  or  weights 

c)  category  number  or  range 

d)  other 

2 .  Standard  items 

3.  Data  collection  or  studies  required 

a.  Monitoring  System 

1.  interrater  reliability 

2 .  testing  of  system  in  organization 

b.  Program/System  Redesign 

1 .  training 

2 .  research 

3 .  continued  development 

4 .  revision 

5 .  maintenance 

6.  customizing  staffing 

c.  Defects 

d.  Cost 

V.  Evaluation  or  Outcomes:  extent  to  which  objectives  are  met 
a.  customer  satisfaction 

1.  CNE/CEO 

2.  supervisory  nursing  staff 

3.  clinical  nursing  staff 

4 .  other 
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b.  Changes  in  pt  nsnignnant,  staffing,  productivity  ate 

c.  Problaas 

1.  lost  workload  (short  LOS,  outpatients,  etc) 

2.  requires  additional  documentation,  forms 

3 .  other 

d.  Benefits 

1.  time  savings  in  data  entry 

2.  types  of  reports 

3.  generation  of  reports 

4.  timely  access  to  information 

5.  easy  to  use 

6 .  other 

f.  Additional  studies  on  tool 
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Appendix  C 


coanTcixllv  Availabl*  Patl»nt  Classification  Svsfn  Somrc^s 
sorf d  Against  8»l»ctlon  Criteria 


Atwork  Corporation  (EXCELCARE) 

Chi  Systems,  Inc. 

Ernst  &  Young 

First  Consulting  Group  (GRASP) 
Gamma  Systems  Services,  Inc. 

James  Bahr  Associates  (ARIC) 
McGladrey  &  Pullen  (UNIQUE) 
McLaren  General  Hospital 
Medical  Management  Planning,  Inc. 
Medicus  Systems  Corporation 
MESH  University  Hospital 
SASHA,  Inc. 

Schick  &  Affiliates,  Inc.  (PACS) 
Van  Slyck  &  Associates,  Inc. 
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